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Diagnosing student problems using the results 
and methods of Physics Education Research 



Edward F. Redish 

Department of Physics, University of Maryland 
College Park, MD 20742-41 1 1 USA 



Learning About Student Learning 

In order to understand what is happening in our classrooms, we have to study our students and their 
responses to our instruction. Physics Education Research (PER) is the subject in which we study how 
students understand (and fail to understand) physics in order to help individual students get over their 
difficulties in leaning physics develop curriculum and materials that are more effective for students. 

Physics education researchers have studied student difficulties with many subjects 1 

• Kinematics • DC circuits 



Dynamics 

Energy 

Heat and Temperature 
Pressure 

Mechanical waves 
Oscillations 
Electric fields 



EM Fields 
Geometrical optics 
Physical optics 
Relativity 
Modem physics 
Quantum mechanics 



This is an impressive list, and much has been learned about the problems students have in learning 
particular topics in physics. Is this sufficient? The list of topics studied sounds like the table of contents 
of a standard text. But we want our students to get more out of a physics class than just “the physics”. 
We want them to develop scientific skills and ways of thinking. We’d like them to - at least in part - 
learn to “think like a physicist”. 

Goals Beyond Content 

In trying to convince students to take physics (and other departments to require that their students take 
physics) we often cite the many skills that can be developed listed as the result of studying physics. 

• Learning how to solve complex problems 

• Learning how to think scientifically 

• Learning how to learn 

• Developing an intuition for the physical world. 

Yet we rarely evaluate our success in achieving these goals. How do we know the extent to which we are 
succeeding? How can we diagnose the state of an individual student or of a class? Most of our 
evaluations focus on content and miss these broader goals. If my students memorize lots of problems and 
can replay anything they’ve seen before but can’t do any problem they haven’t seen — no matter how 
similar. ..? If my students can solve a problem mathematically but can’t tell me what the problem is 
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about, or what the answer means...? Every observant physics teacher has seen these sorts of results. 
Research evidence has now become very compelling to show that learning to solve algorithmic problems 
- even reasonably complex ones - does not imply that our students have developed a sound physical 
intuition or an understanding of even the simplest implications of what they have learned. 

Eric Mazur cites a compelling example in his work with his students at Harvard University. 2 On an 
examination to his algebra-based physics class, he gave the problems shown in figure 1. 

1 . Find the current through the 2 ohm resistor 8 y ^ a 

and the potential difference 
between points a and b. 



2. In the circuit at the left, explain what will happen to 
the following variables when the switch is closed: 
the current through the battery 
the brightness of the bulbs 
the voltage drop across the bulbs 
the total power dissipated 



Fig. 1 : A quantitative and a qualitative problem in DC circuits (from Mazur ; ref 2) 

The average score on the first problem was 75%. The average score on the second was 40%. Students 
found problem 2 much more difficult than problem 1, despite the fact that most physicists would consider 
the analysis of the second problem, the short circuit, much simpler; indeed, parts of it might be 
considered trivial. This study and many others 3 show that students frequently can solve complex 
algorithmic problems without having a good understanding of the physics. 

The apparent inconsistency in what students appear to do suggests we have to think more carefully about 
what we are trying to accomplish. If we want to understand what elements to look for and what to try to 
evaluate we need to understand something about how students think. To do this we need to have some 
model of thinking and learning. 

Often, faculty behave as if they think that a student’s thinking is simply a storage box - a box into which 
we can put knowledge; that we can put knowledge into a student by simply presenting it to them and that 
all we have to know is whether the knowledge is “in there” or not. But cognitive and neuroscientists have 
been documenting for decades that the situation is considerably more complicated than that. 4 

Models of Memory 

Modem cognitive science now has complex and detailed structural information about how memory 
works. In some cases, the process is understood down to the level of neurons. We won’t need all that. A 
few simple principles can help us understand some critical issues. The important ideas are 

1. Memory has two components that our important for our consideration: working (or short term) 
memory and long-term memory. 

2. Long-term memory contains facts, data, and rules for how to use and process them 
(declarative and procedural memory). 

3. Recall from long-term memory is productive and context dependent. 

4. Long-term memory is structured and associative. 
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In order to make these ideas clear, let’s consider an example from student learning of university-level 
physics. An interesting example of productive reasoning appears in a recent paper from Lillian 
McDermott’s group at the University of Washington. 5 The question shown in figure 3 was shown to a 
class of engineering physics students before and after instruction. (Since the interference arises from the 
waves from the two slits interfering with each other ; the pattern would go away and be replaced by an 
almost uniform brightness.) 



When monochromatic laser light is shone 




on a pair of double slits, the pattern 
shown at the right is produced on a 
distant screen. 


util 


wnat wouiu nappen to ine panern 
if one of the slits were covered? 





Fig. 3: A problem in physical optics (from Wosilait et a /., ref 5) 



More than half of the students expected part of the pattern would remain. Some said the left half of the 
lines would remain. Some said every other line would remain. None of the students had actually seen 
this happen, but they created their response from applying some general primitives (reduced cause leads 
to a reduced effect), perhaps supported by their experience with light and shadow. 

The second important idea is that the knowledge in long-term memory is organized by patterns of 
association - probabilities that bringing one bit of knowledge out to working memory will lead to another 
bit of relevant knowledge. The fact that some bit of knowledge or know-how is “in there” doesn’t help 
much if it doesn’t come up when you need it. What’s important is not just what knowledge you have but 
its functionality -how appropriately you access it and how well you can use it. 

The context dependence of the cognitive response has implications for what students do (and how we 
interpret it). Students may hold contradictory models of physical situations at the same time. Problems 
seen as “equivalent” by an expert may not be so seen by a novice. Particularly cues in a problem may 
lead them to bring up one model or another. 



This picture is an oversimplification. 
“Nodes” have structure 
in multiple dimensions. There are 
“metanodes” that control what links 
appear when. 




Links represent 
probabilities of 
association. 

These change 
depending on context. 



Fig. 4: A schematic geometrical representation of a pattern of association 
(schemas) in a knowledge structure. 



Sometimes, individuals may have a particularly robust pattern or cluster of associations - when one item 
is brought up in a particular type of context, other elements are likely to appear as well. In this case, we 
refer to the cluster of knowledge as a schema 6 This is illustrated in the figure above. 
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Examples of schema and their context dependence can be found in Mel Sabella’s thesis. 7 In one example, 
Sabella gave a problem in introductory mechanics in an interview to students taking a graduate class in 
the University of Maryland. The problem was chosen to require a mix of arguments using dynamics with 
forces together with work-energy arguments. Some of the students could do problems with forces or 
problems with energy but had serious difficulties in the context of the given problem. They were unable 
to make the links from their force schema to their energy schema. Figure 5 shows a schematic breakdown 
of the reasoning done by two of these students, one of whom had a well-integrated schema of forces and 
energy, and of a second who was unable to make connections between the two. He occasionally brought 
up the idea of energy, but in each case, the link was a “dead-end”, with no connections to his knowledge 
of energy (at least while he was in the context of the particular problem presented.) 



Student with a unified force/energy schema 
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Fig . 5: Pattern of reasoning of two students in response to a combined 
force/energy problem illustrating use of schemas. 

Our cognitive considerations suggest that in addition to having students master the physics content, we 
also want to consider 

• the extent to which students have a conceptual understanding of the physics 
(the extent to which they see the physics as “making sense”) 

• the “robustness” of their knowledge (their ability to access the correct knowledge and use it 
appropriately) 

• the way the students organize their knowledge (create robust “mental models” of physical 
systems that allow them to reconstruct the knowledge they need when they need it using links to 
appropriate productive reasoning). 

An Example: Mechanical waves and sound 

In order to see how these ideas apply and effect curriculum development, let’s consider an example: 
student understanding of mechanical waves and sound. Waves have the characteristic that the medium 
“recreates” and passes the disturbance onward. This leads to some interesting and counterintuitive 
properties, such as the fact that for linear media such as small amplitude deformations of strings, springs, 
and solids, sound, and electromagnetic fields, the speed of propagation is a characteristic of the medium, 
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not of the “intruding” disturbance. In these systems, the speed of propagation does not depend on the 
shape, amplitude, or frequency of the initiating disturbance. 

Michal Wittmann’s research 8 revealed that many students in our introductory calculus-based engineering 
physics class (18-19 years old) often had a serious misunderstanding of the phenomena. In a series of 
interviews, Wittmannn asked the students how a dust particle floating in the air in front of a loudspeaker 
would move when the speaker played a tone of constant frequency (low - about 10 Hz). A typical 
response is given below. 

It would move away from the speaker ; pushed by the 
sound wave ... / mean , sound waves spread through 
the air , ... the air is actually moving , so the dust 
particle should be moving with that air which is 
spreading away from the speaker. 

The sound wave “hits the particle with ...force. 

(drew figure shown at right) 

Since the sound wave is a pressure oscillation, the particle would oscillate back and forth. This, and 
many other students interviewed felt the particle would be driven away from the speaker. 

This response, and the response to many other interviews and open-ended exam questions, students 
showed a common response. They used a particle-pulse model - a guiding analogy (a pattern of 
association) that a wave crest is like a Newtonian “point particle.” The rest of the wave is ignored. 

Wittmann’s interesting observation was that in a multi-question diagnostic quiz on wave issues, almost all 
students in engineering physics used this analogy in some contexts and the correct reasoning in others. 
These observations are consistent with our schema model and imply that students do not “have the right 
answer or the wrong answer”, but appear to have both answers at the same time . 

Student difficulties are not much affected 
by traditional instruction 

It is well documented in the research literature 1 that student difficulties with physics are often quite 
difficult to change. We see the same thing here. In our traditional instruction, in each of 14 weeks, 
students are lectured to for 3 hours in classes of 100-200, solve about 10 “end-of-chapter” homework 
problems, spend one hour in a small class recitation with a teaching assistant going over those problems 
(mostly in a lecture-like format), and spend 3 hours in a traditional (rather cookbook) laboratory. In a 
preliminary investigation, after traditional instruction only about 25% of the students correctly described 
the dust particle as oscillating. Nearly half described it as being pushed away from the speaker. 

Curriculum to Address Difficulties with Sound 

We modified the traditional class by replacing the recitation by a group-learning session with guided 
discovery worksheets focused on qualitative reasoning 9 These tutorials use a cognitive-conflict model to 
induce conceptual change and require students to make predictions of what they expect to happen before 
they look at data or carry out qualitative experiments. The teaching assistants are trained and serve as 
facilitators, guiding each group’s work by occasional careful questions. We created two worksheet 
tutorials using videos my students Bao Lei and Mel Sabella made at Dickinson College. In these lessons, 
students are asked to account for the motion of a flame observed in the video flickering in front of a 
loudspeaker. Students must resolve discrepancies between their descriptions and their observations and 
develop representations based on their observations of the videos. Two other wave phenomenon tutorials 
were also created and addressed the issues of the particle-pulse model. 



> Dust particle 
is like a block 
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Evaluation: Pre- and Post-Instruction Wave Diagnostic Test 

Our cognitive model and our observations of student response to waves questions strongly suggests that 
students do not have a “true value” of their knowledge of a particular subject. Given a large number of 
questions “seen as equivalent by experts”, an individual student may respond using a variety of models. 
An exam question designed with “cues” or “triggers” to “see if the student really knows it” may give a 
positive response satisfying to both the student and instructor, but may give little information about when 
the students can use their knowledge. 

In order to evaluate the student learning, we developed a variety of questions that offered students 
opportunities to use the particle-pulse guiding analogy. In the different contexts provided, almost all 
students mixed their use of the correct model with the popular (but incorrect) particle-pulse model. We 
measured success not only by the shift of the mean, but by the shift of the distribution. 



A taut string is attached to a distant wall. A demonstrator 

moves her hand to create a pulse traveling toward the wall ~ 

(see diagram). The demonstrator wants to produce a pulse 
that takes a longer time to reach the wall. Which of the actions 
a-k taken by itself will produce this result? More than one 
answer may be correct. If so, give them all. Explain your reasoning. 

a. Move her hand more quickly (but still only up and down once by the same amount). 

b. Move her hand more slowly (but still only up and down once by the same amount). 

c. Move her hand a larger distance but up and down in the same amount of time. ' > 

d. Move her hand a smaller distance but up and down in the same amount of time. 

e. Use a heavier string of the same length, under the same tension 

f. Use a lighter string of the same length, under the same tension 

g. Use a string of the same density, but decrease the tension. 

h. Use a string of the same density, but increase the tension. 

i. Put more force into the wave. 

j. Put less force into the wave. 

k. None of the above answers will cause the desired effect. 



Fig. 6: A multiple-choice multiple-response problem in mechanical waves. 



One example is given in figure 6. This is in a multiple-choice multiple response format. The students are 
instructed to give all the answer that are true. This question specifically demonstrates the students’ use of 
mixed models since many give both responses in which the speed is determined by the medium and by 




# of CM 

answers 



jrt* frf* tail ifW t/nJ m 

PM ~ Pnrri ci ® Prls'cc? Me nlnl Model 

CM - Oonun&uty ConaMsu* (Cccrtcrj Meufid Model 



t** tr*'f 

unfavorable I ** ******** 



PM ^Particle Pulses Aiewal Model 

CM ~ OcfTimunfcy Creusejwm (Crereei) M«n&) M'xW 



Fig. 1: Distribution of pre-instruction and post-instruction responses on the wave diagnostic test showing 
number of questions answered by each student using the particle-pulse and correct models. 
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the hand. They do not appear to notice the contradiction implied by having two ways of determining the 
speed that give different results. (In traditional classes after instruction nearly 75% of the students 
answered this question with both types of responses. 

Probing the mixing of student models 
with multiple choice tests 

Traditional standardized exams in the US are set up as multiple choice in order to allow large numbers of 
students to be tested. These are usually simply scored by class averages of right and wrong. This 
analysis is based on the Spearman assumption (1904) at students have a “true” value, T, and that an 
examination yields a measured value, M, which is the true plus a random variable 10 



An Alternative Model 

The cognitive theory we discussed above suggests that the student has no “true” value, but can most 
appropriately be considered as being in a mixed state. Since the one physical situation in which an object 
can exist in multiple states at the same time is in quantum physics, we use a quantum metaphor in 
developing new mathematical tools. In our proposed model, a student’s state 

|w)= a l \e l )+ a 2 |e 2 ) + 1 X) 

may be mixed plus random. 11 

The evaluation is based on using the traditional hybrid model of test development. (See for example the 
description by Halloun and Hestenes of how they developed the predecessor of the Force Concept 
Inventory, ref. 3.) The process is as follows. 

• Conduct qualitative research to identify student models underlying their spontaneous reasoning 
for a particular content area. 

• Develop a theoretical framework to model the student reasoning process for that particular topic. 

• Develop model-based multiple-choice exams to analyze student model use with quantitative 
measures. 

• Use the results - including the student selection of wrong answers - from the quantitative 
research to facilitate the design of new instructions as well as new diagnostic and evaluation 
tools. 

Mathematical Representations 

In many research studies (see ref. 1), it is found that the large majority of student answers can be 
described in terms of a small number of reasoning patterns. These usually include the correct 
(community consensus) model, one or more moderately consistent (in some sense) alternative 12 model, 
and a pattern of random and inconsistent answers. Bao Lei, a student from Nanjing whom I met at the 3 rd 
US/China/Japan Conference on physics education, has recently completed a doctoral dissertation with me 
at Maryland. In it, he developed a mathematical structure using the quantum metaphor that provides a 
number of useful developmental and evaluation tools. Here I provide only a few examples. A more 
complete description is given in his dissertation. 13 

Assuming for simplicity one alternative student model, we introduce a “model space” with basis vectors 

|e 1 ) = (l,0,0) r |e 2 ) = (0,1, 0) r |e 3 ) = (0,0,1) r 
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We take these to be orthogonal and let \e\> correspond to the correct model, \e 2 > to an alternative student 
model, and \ej> correspond to the student responding randomly (guessing or cueing on irrelevant 
elements). The state of the student is described, in analogy with a quantum amplitude, by the vector 

\ u k)~ 

3 

= =1 
i=l 

where q t is the probability that the student will respond to a relevant problem using the i-th model. 

Model Analysis of a Multiple-Choice Test 

As an example of the process, let us apply the method to a multiple-choice test developed using the 
hybrid model described above, the Force-Concept Inventory (FCI). 14 Five of the questions on the FCI 
address the student alternative conception known as the Force- Motion model (Questions 5, 9, 18, 22, 28). 
We choose the vectors \e\>, \e 2 > , and \e£> to represent the following models: 

• Model 1 . A force is needed to maintain motion. ( incorrect ) 

• Model 2 . An unbalanced force produces a change in the velocity. No unbalanced force is needed 
to maintain a constant velocity. ( correct ) 

• Model 3 . Other ideas and random guessing, (null model) 

To each question, we assign the responses that we expect are associated with a particular model. (This 
expectation needs to be confirmed by detailed interviews in which students consider the test items and 
explain their reasoning.) For the FCI, we get the following assignments. 15 



Question 


Model 1 


Model 2 


Model 3 


5 


a, b, c, 


d 


e 


9 


b, c 


a, d 


e 


18 


a, e 


b 


c, d 


22 


b, c, e 


a, d 




28 


a, d, e 


c 


b 



In a class of N students, the response of the k-th student can be described by the student response vector 6 

2T j 

r k = ( n \ , n 2 ,n 3 ) ai, + n 2 + n 3 = m k = l 9 ...N 

where ti\ is the number of items the student answers using model 1, etc. If we interpret n\lm as the 
probability that the student will use model 1 on this test, then we can write the student state vector as 



I ) — i — ^ 9 V^2 > I 

vw 



Notice that the student state vector is test-dependent. That is, the student state vector represents an 
interaction of the student and the test. The student state vector should not be interpreted as an absolute 
description of some internal property of the student. Since it depends on the test as well as the student, 
the instructor using this method must first decide, based on his or her best judgment of appropriate goals 
and the learning situation, what it is that he or she wants to measure about student learning. As in 
quantum physics, the measurement affects the observation. 
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The Density Matrix 



The quantum metaphor suggests we introduce an additional tool to describe whether the student is in a 
pure or a mixed state: the density matrix. For each student we construct the matrix 



D k =k)( M * 



r T 

U, I 




As it stands, the density matrix for an individual student contains no more information that the student 
scores, m, « 2 , and / 13 . However, when the density matrices for all the students in a class are added, this 
situation changes. 

We construct the density matrix of a class by summing over the individual student density matrices: 



D 



N 






k=i 



If we only summed the individual scores rather than the density matrices, we would lose information 
about how many students are confused (i.e., are in mixed model states). For example, consider the class 
density matrices shown below. In the first case, all the students have the same consistent model (not 
necessarily the correct one). In the second case, half of the students consistently have model 1, 30% 
consistently have model two, and the remaining 20% are guessing randomly. In the third example, a 
number of students have mixed models. This is revealed by the presence of off-diagonal elements. 



;i 0 


0 
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;o.5 
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0- 




3 0.2 


0.3 


0.1- 
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10 0 


ol 




\ 0 


0 


0.2! 




I0.1 


0.1 


0.21' 



All students have the All students have a consistent Students have mixed 

same consistent model. model, but different ones. models. 

The state of the class may then be described by finding the eigenvalues and eigenvectors of the class 
density matrix. 

Traditional statistical approaches that simple sum the scores over the class, effectively only retain the 
diagonal of the density matrix and therefore lose information about the mixing of student models. 
Traditional methods that retain the answer for each student in a large individual question response matrix 
(e.g. factor analysis) attempt to extract model information from the clustering of student responses. This 
only is effective if the students have consistent models. Consider for example a test of m elements given 
to a class of N students in which half the students answer consistently with model 1 and half the students 
answer consistently with model 2. A factor analysis will extract two strong factors. If, however, all of 
the students in the class are using both models, each one 50% of the time, the factor analysis will find no 
strong factor. The assumption of student consistency is built into the method and the information about 
student model mixing is lost. In our method, qualitative research is used to provide the extra information 
as to what models are common. This then allows the use of the matrix of student responses to extract 
information about the extent of student mixed model use. 

Example: FCI Measurement of the Force-Motion Model 

As an example, we gave the FCI as pre and post tests to two sets of classes of students in the first 
semester of our calculus-based engineering course described above. (There was no laboratory in the first 
semester of the three-semester sequence.) One group (labeled traditional) was instructed with TA-led 
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recitation sections, the other (labeled tutorial) replaced these 
sections with the group-learning tutorial environment described 
above. Before instruction, the density matrix for both classes 
were dominated by a single eigenvector that was predominantly 
in the direction of model 1 (force is needed for motion). After 
instruction, the students in the traditional class were in a mixed 
state, with students switching frequently between models 1 and 2 
(Newton 1 model). After tutorial instruction, the class was 
dominated by an eigenvector corresponding to the correct model. 

This is shown in figure 8 at the right. The figure shows the two- 
dimensional subspace of models 1 and 2. The points correspond 
to the eigenvectors. (The vectors themselves are not drawn to 
simplify the diagram. Each point should be thought of as a 
vector in the 1-2 space drawn starting from the origin). The 
distance of the point from the origin represents the fraction of 
students using this particular mixed model represented by the 
eigenvector. The angle of the eigenvector from the coordinate 
axes represents the amount of mixing. A figure of merit can be obtained by measuring the distance 
between the point corresponding to the pre-vector and the “ideal” point (0,1) (all students using the 
correct model all the time). The figure of merit is then the fraction of that distance achieved by the 
instructional method. We refer to this as the instructional efficiency (fraction of the possible gain). For 
these two methods, the tutorial class attained an instructional efficiency of 62% on the force-motion 
concept, while the traditional class achieved 29%. 

This method allows us to make a very detailed statement describing the state of the class before and after 
two different kinds of instruction and to compare them along a particular dimension of student 
understanding using the FCI as a probe. On the force-motion dimension, the traditional class moved from 
being dominated by a nearly pure state of a common alternative conception into a mixed (confused) state 
in which most students used both the correct and incorrect models. The tutorial class made a much 
stronger transition to a nearly pure state using the correct model. 

The statements that we can make using our method provides a much more specific and useful description 
of the class than is possible using traditional methods such as factor analysis which cannot extract these 
details. 17 

Conclusions 

In this paper I have summarized the recent work of the Maryland Physics Education Research Group on 
analyzing student learning in introductory university physics. This work is carried out in a model of 
student thinking and learning basic on principles learned in cognitive science. The model permits us to 
better understand and diagnose the state of both individual students and classes. The most critical 
cognitive principles are: 

• Student responses to a proposed physics context or problem are productive and context 
dependent. 

• Student knowledge is organized into patterns of association or schema. 

In the context of university physics, we conclude: 

• The context that determines a student response includes not only the question or problem and the 
classroom environment, but unobservable (and uncontrollable) elements of the student’s internal 
state. As a result, students can respond to questions that appear equivalent to an expert with 




models of force-motion (dominant 
eigenvector of class density » matrix) 
in response to two modes of 
instruction 
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different, apparently contradictory models. This suggests that an appropriate way to think about 
student models is as if they were able to exist in different states at the same time. 

• Tools developed by analogy with quantum theory (where quantum particles can exist in different 
states at the same time) allows us to describe the state of a class in more detail than traditional 
tools. 

These approaches permit the diagnosing of the state of a class in detail. Having this information can help 

both with the creation and the evaluation of curricular innovations. 
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